Tradeoffs in subjective testing methods for image and video quality assessment

An objective quality estimator for either still images or video should accurately estimate the perceived quality scores of a collection of stimuli. New applications and processing techniques will introduce novel distortions that will need to be quantified in terms of perceived quality in order to confidently evaluate an objective quality estimator. The subjective testing method used to obtain the perceived quality scores affects the accuracy and reliability of the data collected. Two common methods used to collect perceived quality scores are absolute categorical rating (ACR) and subjective assessment methodology for video quality (SAMVIQ). The ACR test method presents stimuli in a random order and uses a coarse resolution rating scale for evaluation. The SAMVIQ test method allows the observer to freely view several stimuli multiple times and uses a fine resolution rating scale for evaluation. Ease of implementation typically influences the adoption of ACR over SAMVIQ, since ACR accommodates more stimuli per testing session. This paper investigates the tradeoffs of these two subjective testing methods using three different subjective databases that have perceived quality scores corresponding to the ACR and SAMVIQ test methods. Results are 1) the fine resolution rating scale used by SAMVIQ is superfluous, 2) SAMVIQ scores have greater accuracy than ACR scores for the same number of observers (on average 30% fewer observers were required for SAMVIQ than ACR for the same level of accuracy), 3) SAMVIQ scores better differentiate stimuli than ACR scores, and 4) the consistency of categorical ratings between ACR and SAMVIQ is lower for databases when stimuli are more difficult to distinguish in terms of perceived quality. Increasing the number of observers for ACR generates more accurate scores, competitive with the accuracy found with fewer observers using SAMVIQ. Despite the evidence promoting the use of SAMVIQ to obtain perceived quality scores, scores obtained with ACR predict those obtained with SAMVIQ when the stimuli are easier to distinguish in terms of perceived quality.